User interface for removing an object from a display

ABSTRACT

A digital system that may be used by children two years old and older. The digital system is contained in a child-proof case and has an upward-facing display with a touch-sensitive screen that is within easy reach of a child. Other I/O devices include proximity and motion sensors and a microphone, and there is also a loudspeaker. When a proximity sensor senses someone in the neighborhood of the system, it displays images on the display. A child may manipulate the images by touching them on the touch screen. Manipulations include selecting an image by touching it, “dragging” the selected image by moving the finger touching the image across the screen and “dropping” the image by lifting a finger from it, moving a selected image by touching another location on the screen and thereby causing the selected image to move to the touched location, removing an image from the screen by “throwing” it, i.e., moving it above a threshold speed, and modifying the image by tapping it twice and then moving the finger in a horizontal or vertical direction on the screen. The direction in which an image is thrown may further determine what the thrown image is replaced with. The manipulations are used to in activities such as shape matching, puzzle assembly, assembly of a face out of parts, and hide-and-go-seek.

REFERENCE TO RELATED PATENT APPLICATIONS

The present patent application claims priority from U.S. ProvisionalApplication 60/057,117, Slavoljub Milekic, Child-friendly DigitalEnvironment, filed Aug. 28, 1997.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The invention relates generally to digital systems and more particularlyto digital systems that are adapted to use by children two years old andolder. One aspect of such a digital system is a graphical user interfacethat requires neither typing skills nor fine visual-motor coordination.

2. Description of the Prior Art

Ever since interactive computer systems became available in the 1960's,they have been used to educate and entertain children. Entire industriesdedicated to interactive games and educational software have arisen andthe Internet has literally made the whole world available to a childwith access to a personal computer that is connected to the Internet.Educational uses of the computer have ranged from employing it as a pageturner and exercise checking machine through using the fact that it canbe programmed to teach analytical thinking and problem solving skills.For an example of the last kind of application, see Seymour Papert,Mindstorms, Basic Books, 1980.

Until now, the computer as used for the education or entertainment ofchildren has had the same basic form as the computer as used in theworkplace: the display sits on a desk and has a vertically-mountedscreen and input to the computer has been by way of a keyboard and apointing device such as a mouse that sit on the desk with the display.The graphical user interfaces have generally been based on at least twoout of three assumptions: the user can type, the user can read, and theuser has the fine motor coordination necessary to manipulate thebuttons, sliders, and icons typical of modern graphical user interfaces.

The orientation of the screen, the input devices, and the graphical userinterfaces together render a standard computer unusable by children ofpre-school age. Such children are too short to see the display or reachthe keyboard and mouse, they cannot read, they cannot type, and even ifthey could reach the keyboard and mouse, they do not have the fine motorcoordination necessary to use the graphical user interface. Moreover, astandard computer is not child-safe: it has an exposed power cord andother cords connecting components such as the keyboard and mouse to theCPU.

What is needed if small children are to be able to take care of theeducational and entertainment opportunities offered by the computer is adigital device which has been rendered child-safe and which has a userinterface that permits direct manipulation of objects in the display andrequires neither literacy nor typing skills nor fine motor coordination.It is an object of the present invention to provide such a digitaldevice.

SUMMARY OF THE INVENTION

The child-friendly digital system of the invention differs both in itsphysical aspect and in its graphical user interface from standarddigital systems. In its physical aspect, the child-friendly computersystem is contained in a toddler-proof case that rests on the and has atouch-sensitive screen that is within easy reach of a toddler. Imagesare displayed on the touch-sensitive screen and the child-friendlydigital system responds to touches on the display by altering thedisplay. Other features of the physical aspect include an upward-facingdisplay, sensors for sensing the presence of the child and motion abovethe display, a microphone for receiving voice inputs, and a loudspeaker.The child-friendly digital system has not cords or other appendages. Inone embodiment, the digital system is portable; in another, it is afixed unit.

The graphical user interface for the child-friendly digital system isbased on manipulating an image on the touch-sensitive screen by touchingthe image directly. If an image is movable, touching the screen at theimage selects the image for moving; moving the touched point within aselected image causes the image to move with the touched point, thuspermitting the image to be dragged. An image that has been selected formoving may also be caused to move to another location by touching apoint elsewhere on the screen. The touch causes the selected image tomove to the selected point. If the screen is touched at two or morepoints simultaneously, the image moves to a point between the touchedpoints. If the image is dragged at a speed above a threshold velocity,the image is “thrown away from” the display and may be automaticallyreplaced by another image. Depending on the direction in which the imageis dragged, the image may be replaced by one of the same kind or one ofa different kind. If an image is tapped twice in short succession, withthe finger remaining down after the second tap, the image is selectedfor modification. Moving the finger on the screen when an object hasbeen selected for modification causes the object to change size indirections that depend on the direction of motion of the finger on thescreen. The actions permitted by the graphical user interface are usedto implement activities including shape sorting, puzzle assembly,hide-and-go-seek, and what might be called a digital picture book.

Other objects and advantages of the invention will be apparent to thoseskilled in the arts to which the invention pertains upon perusing thefollowing Detailed Description and Drawing, wherein:

BRIEF DESCRIPTION OF THE DRAWING

FIG. 1 shows a portable embodiment of the child-friendly digital system;

FIG. 2 shows a non-portable embodiment of the child-friendly digitalsystem;

FIG. 3 shows a display from a first shape-matching activity performed onthe child-friendly digital system;

FIG. 4 shows a display from a second such shape-matching activity;

FIG. 5 shows a display from a puzzle assembly activity;

FIG. 6 shows a display for an activity in which a face is assembled outof components;

FIG. 7 shows a display for a hide-and-go-seek activity;

FIG. 8 shows how activities using the child-friendly digital system maybe combined with activities using actual objects in the child'senvironment;

FIG. 9 shows how the child-friendly digital system may be used to viewpictures;

FIG. 10 shows behavior of the display when the display is beingsimultaneously touched in two or more locations;

FIG. 11 shows an arrangement of presence and motion sensors in apreferred embodiment;

FIG. 12 shows a first portion of a script for the face assemblyactivity;

FIG. 13 shows a second portion of the script;

FIG. 14 shows a third portion of the script; and

FIG. 15 shows a fourth portion of the script.

The reference numbers in the drawings have at least three digits. Thetwo rightmost digits are reference numbers within a figure; the digitsto the left of those digits are the number of the figure in which theitem identified by the reference number first appears. For example, anitem with reference number 203 first appears in FIG. 2.

DETAILED DESCRIPTION

The Detailed Description begins with an analysis of the kinds of changesthat must be made in a digital system if it is to be usable bypre-school children, continues with a description of the physicalconstruction of such a system, and then describes the graphical userinterface for such a system. Finally, the Detailed Description providesa detailed disclosure of the implementation of important aspects of thegraphical user interface.

What Needs to be Changed to Make a Digital System Child Friendly

There are three major areas which need to be addressed in making digitalsystems child-friendly. They can be loosely defined as changes in a)location, b) mode of interaction and c) content structure. Each of thesechanges will be briefly described in the following paragraphs.

Change in location. Although it seems trivial at first, change oflocation of objects is the first indicator of the psychological changeof domain perception. Just moving the computer from the desk to thefloor makes it more accessible to children but also indicates to themthat the computer is a legitimate part of their environment. Of course,modern computers would hardly survive this change, because they were notbuilt with children in mind. Moving computers to the floor would alsomean making them: at least as child-resistant as any good toy. As simpleas it is, change in location also implies a host of other changes in thedesign of child-friendly digital devices. First, a child-friendlydigital system should lose all of its appendages and the cords thatconnect them. This means getting rid of the power cord, the keyboard,the mouse, and their cables, and making the image-displaying partself-standing (battery operated or with a concealed electric cord).Putting the display on the floor also means change in the orientation ofthe viewing surface from perpendicular (where the child had to look up)to a more physiological upward-facing angle.

Change in mode of interaction. The change in the mode of interaction isnot only dictated by the change in location, but is also necessitated bythe inadequacy of the keyboard and the mouse as input devices forchildren. Both devices depend on possession of special kinds ofknowledge and skills, not readily available to children. The keyboardrequires not only competent writing (and typing) skills but alsoknowledge of specific vocabulary and its use (for example, that typing“exit” will end the current game). This does not mean that in achild-friendly interface environment keys would be banned fromexistence, but only that their number, size and function woulddramatically change.

The mouse suffers from similar shortcomings. Not only is it inherentlyabstract (moving the mouse moves something in the display, but notalways . . . ) but it also involves fine visual-motor coordination. The‘folders’ displayed in a typical modern graphical user interface areapproximately ¼ inch square and it is within this range that the childhas to coordinate the movement with the ‘click’ (sometimes even‘double-click’) in order to make something happen. The size of theinterface elements is not the only problem. One could easily increasethe size of typical ‘buttons’ on the screen and use another input device(like the touch-sensitive screen), but the problem of interaction stillremains. The adult-designed ubiquitous ‘desktop’ metaphor with its filesand folders, and subfolders and ‘windows’ (on the desktop?!) is hardly atypical child's handy metaphor. The necessity for changing not only theinput devices but also the way the digital information is renderedaccessible is the topic of the next section.

Change in content structure. The change in content structure does notmean change in content per se, but rather change in the way the contentis organized and presented to the child. To an illiterate person (or achild) all the ‘folders’ on a computer display look pretty much thesame. Thus, in a child-friendly digital environment the indicators ofcontent should be clearly distinct visually and represent familiaraspects of the child's experience. However, this is the most superficialchange necessary. There are other aspects of children's activity thatcall for more radical changes. These are a) making the information(content) manipulable, and b) making the content structure compatiblewith the child's social environment.

-   -   a. The idea that making the information experientially        accessible to the child leads to more efficient knowledge        transfer has been around for some time but has not been        consistently implemented in the area of child/computer interface        design. Thus objects should be made manipulable in the way which        makes sense to the child and provides feedback which can        compensate for the unavoidable impoverishment of sensory input        in comparison to equivalent real-world manipulations. Several        ways of achieving this goal will be discussed in the following.    -   b. Making the content structure compatible with the child's        social environment means that it readily supports the social        interactions a child is likely to engage in: interactions        (playing) with peers and interactions with adults (educators,        parents). Originally, computers (as the ‘personal’ part of ‘PC’        implies) inhibited interaction and collaboration between        individuals. It is only recently, with the popularization of the        Internet (which was made possible by a friendlier interface        design) that the importance of supporting collaboration is being        discovered again, and that different software products        supporting it are boasting three-digit return rates. Although it        is possible to imagine similar solutions for children's computer        environments, it is important to realize that a child-friendly        environment has to support social interactions more concretely,        and respond appropriately to simultaneous and possibly divergent        inputs on the same physical unit.        Hardware for a Child-Friendly Digital System: FIGS. 1 and 2

As described above, moving the digital system from the desktop to thefloor leads to dramatic changes in design. The system loses separateinput devices such as the keyboard and mouse and is reduced to thedisplay unit. Ideally, the display itself should be compact, mobile, andwith an upward-facing touch-sensitive viewing surface It should also berugged, scratch-resistant and use a built-in power source. The unitshould also have ample storage capacity and a way to quickly access,modify and update stored information. Digital systems with some of theabove characteristics are already available. They may be found in ATMmachines and in kiosks for finding locations in superstars, malls,airports, or even museums. None of them, however, is designed to sit onthe floor or has a graphical user interface that a preschooler can use.

FIGS. 1 and 2 show two versions of the hardware for a child-friendlycomputer system. FIG. 1 shows a portable child-friendly digital system101 and FIG. 2 shows a stationary version. As shown in FIG. 1, portabledigital system 101 is based on a laptop computer (not shown). Digitalsystem 101 has a touch-sensitive active matrix LCD display 111, and thesuggested minimum configuration for the laptop is a, 1.6 G hard drive,32 MB RAM, a 28.8 modem, a 16-bit sound card, and a built-in infra-redcommunications port. The laptop components are fitted into case 103 ofheavy duty molded plastic with rounded edges 105. Display 111 forms thetop of case 103. Also housed in the case are microphone 107, speakers113, and motion/proximity sensors 109, all of which are connected to thelaptop computer. Display 111 serves both as an input device and anoutput device, microphone 106 and sensors 109 serve as additional inputdevices, and speakers 113 serve as output devices for the laptop.Sensors 109 permit the system to sense the presence of the child and torespond with verbal prompt, movement, or sound. Sensors 109 are builtinto the top edge of the child-friendly digital system and consist of aseries of sensitive motion and proximity sensors with varying fields ofview. The ‘points’ of view of different sensors divide the space infront of the unit into a virtual grid, which makes it possible not onlyto detect the presence of the child, but also to incorporate simplegestures (for example, moving of the hand from left to right) intointeraction with the software. As shown in FIG. 1, display 111 is set upfor a shape-matching game: it displays two objects, a circle 115 and acutout 117 into which circle 115 fits.

The stationary version 201 of the digital system is better suited formuseums and day-care centers. It has the same technical characteristicsas portable system 101 but is based on a desktop computer with atouch-sensitive monitor 205. The system is configured as shown in FIG.2. The CPU is hidden inside a structure 203 encasing monitor 205. Thewhole structure 203 is built in such a way that it can support anaverage adult's weight and should have a non-slippery surface and nosharp corners or edges. Structure 203 further contains a speaker 213 andmay also contain a microphone and motion and proximity sensors. It mayalso have a light shade 211 to make the display on monitor 205 moreeasily visible to users of system 201. Structure 203 finally provides aplay area upon which selected objects 209 can be displayed. If theobjects are the same as those being displayed in display 205, system 201will help children begin to understand how a three-dimensional object isrelated to its two-dimensional representation.

Child-Friendly Graphical User Interface

The graphical user interface in system 101 or 201 is based on thetouch-sensitive display, the motion and proximity sensors, and a voicerecognition system that is made using microphone 107 and custom-made orcommercially available voice recognition software such as DragonNaturally Speaking from Dragon Systems, Inc. or Via Voice from IBMCorporation. The touch screen is employed as the primary user inputdevice for a number of reasons. Pointing to and touching an item are themost natural ways of indicating its selection, and require no trainingeven in very young children. Touch screens are very durable, have nomoving parts, and require almost no maintenance. Since they aresuperimposed over the viewing surface, they demand no additional space.A host of studies on adults (summarized in Sears and Schneiderman, “Highprecision Touch screens: design strategies and comparisons with amouse”, International Journal of Man-Machine Studies 34, pp. 598-613,1991) indicate that Touch screens are the fastest pointing devices.However, if used for the selection of very small targets (less than 10mm in diameter), they are also the ones with the highest percentage oferror rates. The results were partially caused by the low resolution ofolder Touch screens and the returning of multiple pixel locations by thetouch screen hardware. In the past several years both the increasedresolution of touch screens and the software-implemented strategies forstabilizing the touch location have reduced touch screen error rates andbrought them in line with those of the mouse. It is worth noting thateven with the older touch screens there was no difference in error ratesbetween the touch screen and the mouse in conditions where largerselection targets were used. A pilot study conducted at the HampshireCollege Cognitive Development Lab has shown that even children as youngas 2 years find the use of a touch-sensitive screen intuitive and easy.Furthermore, their performance on a simple visual mapping task was quitegood, possibly as the result of the decrease in cognitive loadassociated with the interface.

While the touch screen makes a child-friendly graphical user interfacepossible, it is not sufficient by itself. Since preschoolers cannot readand do not have the fine motor coordination necessary for standard GUIs,the GUI had to be redesigned to employ interactions that were easy forthe preschoolers. The interactions included the following:

Selection: Selection of an object such as circle 115 in display 111 iscarried out by touching it. There is no traditional highlighting of theselected object (necessary for the mouse input) because of the existinghaptic feedback. However, since all of the objects have defined ‘anchor’points (119 in object 115) which are used for ‘dragging’ action, thereis often a small movement of the selected object as it aligns its anchorpoint with the touch point. Selection may be further indicated by avisual “lifting” of the selected object (i.e., a discrete shadow isadded to the object when it is selected and/or by a discrete auditorysignal when the object is selected or released.

Moving objects: Selected objects can be dragged by moving the fingeracross the screen, and ‘dropped’ by lifting the finger. If objects aredropped over the appropriate slot (117 in FIG. 1) (which usuallycorresponds in size and shape to the object being dropped), there is asuitable form of visual and acoustic feedback.

In addition to the traditional mouse-supported actions, there are threemore types of interaction supported in the graphical user interface:pointing to a location, throwing the object, and pushing the object.

Pointing to an object: Pointing to a location consists of simplytouching screen 111 at the desired location. If what was last selectedwas an object, the touch may cause the object to move to the locationthat was touched. If what was last touched was a slot, the object thatfits the slot will move to the location that was touched.

Throwing an object: The throwing action is executed when the speed atwhich an object is dragged across display 111 exceeds a threshold speedwhich corresponds more or less to the speed of the natural throwingmotion. When the threshold is exceeded, the ‘thrown’ object willcontinue to move in the same direction even when the finger is liftedoff the screen. One use of throwing is to remove an object from thedisplay.

Pushing: When the child's finger is moved along display 111 and touchesthe side on an object, the object starts moving in the same direction infront of the finger. The motion is terminated when the finger is liftedup from the screen or when it stops moving.

Object modification: Some objects may be modified by the child. When thechild taps the object twice and doesn't raise its finger after thesecond tap, the object is surrounded by a red outline. If the child thenmoves its finger up or down on the screen, the object changes size inthe vertical direction; if the child moves its finger sideways on thescreen, the object changes size in the horizontal direction.

Voice recognition: The voice recognition mode of interaction allows thechild to perform simple actions and navigation using oral commands like“go”, “no”, “more”, “new”, etc.

Hand gestures: Simple hand gestures (without touching of the screen) canbe used for interaction in certain contexts. For example, moving a handin horizontal direction in front of the screen may cause depictedobjects, images, or pages to be ‘flipped’ in the appropriate direction.Another example is that of ‘zooming’ into the picture by just moving thepalm of the hand closer to the screen (that is, to the proximitysensor).

Using the Interactions to Make Activities

The following portion of the description shows how the interactionsdescribed above may be used to make activities for preschoolers.

Matching: FIGS. 3-5

FIG. 3 shows how objects like circle 115 may be selected by touching andonce selected, they may be dragged to another location by moving thefinger across the screen. When selected (touched) the object aligns itsgeometric center 119 with the tip of the finger, thus indicating theselection. When dragged to within close proximity to the matchingcutout, the object is ‘pulled’ into the cutout, as shown in inset 305.The size of the area which triggers the ‘pulling’ of the object(indicated by dotted line 303 above, not visible on the actual screen)is adjustable, making it possible to compensate for the lack of finecoordination (for example, in very young children) during matching. Ifthe child taps on cutout 117, matching object 115 will move on its ownto cutout 117. Side cutouts like 309 indicate possible navigationalpaths. Touching side cutout 309 leads to another screen, often with anew type of activity.

As shown in FIG. 4, the simple matching of FIG. 3 can be expanded tomake a matching game that works like a shape sorting box. In display401, objects can be selected and moved anywhere on the screen. If placedover or in close proximity to the matching cutout the object will be‘pulled in’. In display 401, the triangle and the square shapes arealready in their matching, cutouts. Navigation is carried out via theside cutouts. Touching the left cutout 309 leads to the previous screen,while touching the right cutout leads to a new screen. Display 401 canthus provide a child with an entire collection of shape sorting boxes.When a child drag an object, the loudspeaker emits by a ‘scratching’noise, and on a successful match, the loudspeaker emits a clickingnoise.

Assembling a jigsaw puzzle is also a shape-matching activity, and FIG. 5shows how the child-friendly digital system can be used to provide anactivity very similar to assembling a puzzle. The activity starts andends with the assembled puzzle, as in inset 507. Touching the assembledpuzzle 507 leads to random scattering of puzzle pieces all over thescreen. As in previous activities, pieces can be selected and movedacross the screen using the finger. Selecting a piece like 503 andplacing it over the proper position 505 ‘snaps’ the piece into itsplace. The active piece (the one being selected or moved) always movesinto the topmost layer of display 111 and is thus shown above all otherpieces. By touching the navigational cutouts 309 on either side of thescreen it is possible to completely bypass this activity. Leaving theactivity while the puzzle is half finished leads to automatic assemblyof the missing puzzle pieces.

Making Faces: FIG. 6

At the start of the activity, the face parts shown at 601 are randomlyselected and shown on display 111. In the beginning facial elements thatoccur in pairs are matching and symmetrical. To change the face, thechild uses its finger to move each part to a desired location. The childmay also use the throwing interaction described above to replace a facepart with a new face part. For example, an ear of the type shown at 607may be replaced by one of the type 617. The exchange is carried out bythrowing” the selected part toward the bottom part of the screen. Whenthis is done, a different part of the same class appears, for example,ear 617. In order to allow creation both of symmetrical, natural lookingand ‘strange’ faces ‘throwing’ of the symmetrical parts is regulated inthe following fashion: If both parts are of the same kind (for example,two blue eyes) throwing away of one part results in random replacementby a different kind (for example, with a green eye). This allows for themaking of ‘hybrid’ faces. However, if the user wants to create adifferent symmetrical face, ‘throwing’ away of an element in amismatched pair will result in the replacement with the appropriateelement (that is, throwing away of a blue eye in a blue-green pair willresult in appearance of another green eye). Double-tapping (and holdingthe finger down) on any face part puts the face part into a ‘modifiable’mode indicated by a red outline around the part, as shown at 619. Whilethe part is in this mode moving the finger up or down (623) across thescreen changes the part's height, while moving the finger left or rightchanges its width.

Hide and go Seek: FIGS. 7 and 8

FIG. 7 shows how objects in the child's environment can be included inthe activities the child uses the child-friendly digital system for. Theobjects depicted in display 701 (flowerpot 705, duck 709, basket 711,and bear sticker 703) correspond to actual objects in the child'senvironment. Both real objects and objects on the screen can be used fora guessing game. Using his or her finger, the child can drag sticker 703around and hide it behind other objects. Inset 713 shows how the childhides the sticker by pushing part of it behind an object. When the childlifts its finger from the sticker, the sticker is “pulled” behind theobject, so that it is no longer visible. Once the sticker is hidden itis possible to play the guessing game—when a child taps on the objectthat hides the sticker with its finger, the sticker slides out with anappropriate sound effect. Tapping on other objects elicits a differentsound, indicating that the sticker is not hidden behind that object.

FIG. 8 shows at 801 how the child-friendly digital system may be used torepresent real objects already known to the child and how it therebyinvites the child to explore the representations in display 111 in thesame fashion that it explores the real objects. Here, the real objectsare toy houses 805 and 803 and a real sticker like 703. A child may hidethe sticker under one of the toy houses and then find the sticker bymoving the houses. Display 111 has representations 807-811 of the samereal houses and a representation of sticker 703. The child may hide thesticker as described above, and can find it by tapping therepresentations of the houses.

Using Throwing to Navigate within and Between Categories: FIG. 9

As is clear from the activities already discussed, a child-friendlydigital system can provide a child with many different versions of thesame activity as well as with many different kinds of activities. Thechild-friendly digital system must thus provide the child with a simpletechnique that permits it to both navigate among classes of activitiesand versions of activities in a class and to select a one of theactivities. In a preferred environment, this is done with throwing. Toget another version of the present object or activity, the child throwsthe present one to the left or the right; to get a different class ofobject or activity, the child throws the present one up or down. FIG. 9shows how this works with pictures belonging to different categories.The pictures have been divided up into the categories “deer” (901, 905,903), “faces” 907, and “outdoors” (not shown). Presently, deer picture901 is being displayed on display 111. If picture 901 is thrown to theleft, deer picture 903 is displayed; if picture 901 is thrown to theright, deer picture 905 appears. There may be a predetermined order ofpictures or the next picture to be displayed may be selected at randomfrom the group of pictures belonging to the same category. The child canthus use throwing to the right or left to explore a category. If thechild wishes to examine a new category, the child throws present picture901 up or down. In FIG. 9, throwing current picture 901 down causes apicture belonging to faces category 907 to be displayed; throwingcurrent picture 901 up causes a picture belonging to the “outdoors”category to be displayed; to see other pictures within the category thechild throws the current picture to the left or right.

The same mechanism may be used with games. For example, throwing theshape sorting display of FIG. 4 to the left or right may result inanother shape sorting game being displayed, while throwing it up or downmay result in a puzzle or a face assembly game being displayed.Moreover, the activities within a category may be ordered as required bythe pedagogical purpose of the category. An example might be picturesthat were ordered by increasing distance from photographic realism.

Collaborative Activities: FIG. 10

FIG. 10 shows the behavior of display 111 when it is being touched attwo points at once. At 1001, a child is touching an object 1003, and itmoves with the child's finger, as previously described. At 1005, anotherchild touches display 111 and object 1003 moves midway between thepoints at which the children are touching the display. At 1007, thefirst child ceases touching the display and object 1003 moves to thepoint at which the second child is touching the display. If more thantwo children touch the display at once, object 1003 moves to a pointthat is within the points being touched. This behavior of display 111permits collaborative manipulation of an object.

DETAILS OF A PREFERRED EMBODIMENT

In what follows, details of an implementation of the invention will bediscussed. The discussion is based for the most part on animplementation made for use in an art museum.

Implementation of Display 111

Display 111 is implemented in a preferred embodiment using a resistive,capacitive, or surface-acoustic wave touch screen. Such touch screensare available from vendors such as MicroTouch Systems, Inc., ELO,PixelTouch, or Keytec. The type of touch screen will of course determinewhat kinds of interactions are possible. For example, thepreviously-described behavior of a selected object when display 111 isbeing touched at more than one point simultaneously makes use of thefact that when a resistive touch screen is touched at more than onepoint simultaneously, the position that the touch screen driver reportsto the CPU is the average of all of the positions. Some touch screens,for example those that use surface acoustic waves, can detect pressureas well as position, and this fact can be used in the interactions. Anarea of current experimentation is using a pressure sensitive screen toimplement “3-D” interactions.

Implementation of Motion and Proximity Sensors 109: FIG. 11

FIG. 11 shows a presently-preferred arrangement 1.101 of motion andproximity sensors. There are three sensors: a single proximity sensor1103 located at the middle of the top of the display and a pair ofmotion sensors 1105(a and b), one of the pair being located at each endof the top of the display. The sensors are constructed usingcommercially-available photo- or heat-sensitive cells, with input fromthe sensors going via a commercially-available ADBIO box by BeehiveTechnologies for Macintosh computers. Of course, a custom-designed databox may also be used. Proximity sensor 1103 has a wide field of view,while motion sensors 1105 have narrow fields of view. The motion sensorsare adjusted so that their fields of view intersect 40-60 cm. above thecenter of display 111.

Proximity sensor 1103 detects the presence of an object near thechild-friendly digital system. One way of using the proximity sensor isto detect the presence of a child near the system. When the child isdetected, the system may make objects appear on display 111 or even gointo an “attract mode” specifically designed to attract the child'sattention. Another way of using it is to have objects in the displayrespond to the degree of closeness of an object detected by theproximity sensor. For example, a figure on the display might respond tothe approach of a child's hand to the display by moving out from underthe hand and thereby appearing to “run away” from it. Another examplewould be the “zoom” effect mentioned earlier.

Motion sensors 1105 together detect the direction and velocity of motionabove display 111. Motion sensors 1105 can be used in the same fashionas proximity sensor 1103 to detect the presence of a child. They canalso be used as a source of inputs that influence the behavior of thedisplay. For example, throwing could be implemented as a response to arapid horizontal or vertical motion of the hand above display 111.Another use of motion sensors 1105 is to control a figure on display 111that responds to the motion sensed by the motion sensor. For example, ifthe child moves its hand from left to right, the figure does the samewith its hand.

Programming the Child-Friendly Digital System

A prototype of a child-friendly digital system has been implemented in aMacintosh® computer manufactured by Apple Computer, Inc. The prototypehas been programmed using the Supercard® programming environmentmanufactured by Allegiant Technologies, Inc., 9740 Scranton Rd., Suite300, San Diego, Calif. The Supercard programming environment wasdeveloped to create interactive displays. An interactive display iscalled a project in the Supercard environment; each project is asequence of one or more windows. Within a window, a sequence of one ormore cards may be displayed; a card may have associated with it abackground design, and a number of objects such as text, graphics, orelements such as buttons and menus. Also associated with each project isa script written in the Supertalk™ scripting language. The code in thescript is executed in response to events such as a touch on display 111.Details concerning the Supercard programming environment and theSupertalk scripting language may be found in Sean Baird, et al.,SUPERCARD User Guide, Allegiant Technologies, Inc., 1996, and Ken Ray,et al., SUPERCARD Script Language Guide, Allegiant Technologies, Inc.,1996. Both of these documents are hereby incorporated by reference intothis patent application. In the following, the script language code forcertain of the interactions will be explained in detail.

In the Supercard programs used to implement the activities describedabove, a project consists of a number of different kinds of activities,for example, shape matching, hide-and-go-seek, jigsaw puzzles, and faceassembly. There is a window corresponding to each class of activity anda card corresponding to each version of the activity. Graphical objectsare used for the shapes, slots, puzzle pieces, face parts, and the like.Behavior of the graphical objects is controlled by scripts.

Details or the Scripts for the Activities: FIGS. 12-15

The scripts in the following are for an implementation of the faceactivity shown in FIG. 6. This activity employs selection of a componentof the face by touch, dragging the component, replacing a component withanother component by “throwing” the component, and modifying acomponent. The following detailed explanation will explain thoseinteractions, and those skilled in the arts to which the inventionpertains will easily understand from the explanation of thoseinteractions and from the documentation for Supercard how to writescripts that perform the interactions required for the other activities.

FIG. 12 shows a graphic for a component of the face at 1201 and thefirst portion of the code for the MoveMe event. As shown at 1201, eachgraphic has a name 1203 and a number 1205. The name Leye5 1203 indicatesthe class of component, namely an eye, whether the eye is a left eye orright eye (here, a left eye, indicated by L), and which of a number ofsets of eyes it belongs to (here, set 5). The script associated withcomponent responds to a MouseDown event 1207, which is an indicationprovided by the SuperCard system that the user has touched screen 111inside the component. The script performs a single action in response tothe event, namely, it causes a user-defined event called moveMe tooccur.

The code that is executed when moveMe occurs is shown at on MoveMe1215(A) and (B) in FIGS. 12 and 13. In broad terms, on MoveMe 1215performs the interactions that occur when the user selects a componentof the face by touching it, moves the component by dragging it, obtainsa new component by throwing the component, or modifies the component bytouching it twice and then moving the point at which the component istouched within the component. Beginning in FIG. 12, declarations 1217specify that MoveMe uses four global variables that it shares with otherportions of the script. StartTime contains a timer value; OrigPositioncontains the value of the position of the graphical component when themoveMe event occurs; CurrPosition is the current position at whichscreen 111 is being touched; and CurrPict is the component of the facewhich is presently selected for movement.

The code of 1215A then saves the current time (provided at regularintervals by the SuperCard system) in StartTime at 1219, sets thelocation of the target (that is, the component currently being touched)to the present location of the mouse at 1221, puts the target's shortname into CurrPict (1223), and puts the target's location intoOrigPosition. It is the line of code identified at 1221 that moves thecenter of the target to the point touched by the user, thereby providingfeedback indicating that the touch has selected the target for moving.

Continuing with FIG. 13, portion 1301 of MoveMe implements dragging thecomponent; portions 1311 and 1317 implement throwing; portion 1319 doesresizing. Considering portion 1301 in more detail, portion 1301 is aloop which is executed from the time the user touches the componentuntil the time the user ceases to touch the component. The steps of loop1301 are the following;

-   -   The current location of the touch is saved in a local variable,        OldPosition;    -   the target is moved to the current location of the touch;    -   there is a wait of two clock ticks;    -   the current touch position is placed in the global variable        CurrPosition;    -   if the user is no longer touching screen 1111, the loop is        exited; otherwise it is repeated.

Portions 1311 and 1317 implement throwing. The position variablesOldPosition and CurrPosition each contain two values, item 1, which isthe x coordinate, and item 2, which is the y component. In loop 1301,OldPosition is set a little more than two clock ticks beforeCurrPosition; consequently, the velocity with which a component is movedcan be determined from the distance between the position specified inOldPosition and the position specified in CurrPosition (1312). In thisembodiment, the velocity threshold is a distance of 2 of the distanceunits established by SuperCard. If the distance between the positionvariables is greater than that, a throw has occurred. In 1311, thethrowing motion is to the left; in 1317, the cases where the throwingmotion is to the right, up, or down are dealt with. Here, we need onlydescribe 1311 in detail. As shown at 1313, first the user-definedthrowLeft condition 1313 is raised; this gets rid of the part beingthrown. Then the user-defined condition GetNewPart 1315 is raised; thisgets the replacement part. Finally, MoveMe code 1215 is exited.

The code that is executed when these conditions are raised is shown inFIG. 15, with the code for throwLeft at 1501 and the code for GetNewPartat 1507. The code for throwleft, on throwLeft 1501 simply moves thecurrently-selected component (indicated by the target) to a positionwhich is off screen 111 to the left, thereby removing thecurrently-selected component from the display. on GetNewPart 1507 ismore involved, since it must replace the thrown part with another of thesame type, and if the thrown part is a part such as an eye that comes inpairs, on GetNewPart must respond to a member of a mismatched pair thatis thrown by providing a member that matches the non-thrown member andmust respond to a member of a matched pair that is thrown by replacingit with a randomly-chosen member.

The first thing GetNewPart does is rename the current target“thrashedOne”. Then at loop 1509, GetNewPart 1507 deals with the problemof pairs of parts. As mentioned above, the parts in a pair may eithermatch or not match. Conceptually, each part belongs to a class of parts,such as eyes, noses, etc. Within the class, the part has an ID number,and if it is a part that comes in pairs, it has an indication whether itis the left or right member of the pair. In the preferred embodiment,this information about the part is encoded in the part's name. Forinstance, in the part name Leye5, eye indicates the class name, 5 thekind of eye, and L that the eye is a left eye.

In loop 1509, GetNewPart examines the name of each part in the displayin turn. It keeps going until it finds a part that belongs to the sameclass as the current target or it has examined all of the parts. Asshown at 1511, if it finds a part that belongs to the same class and hasthe same ID as the current target it exits loop 1509. This deals bothwith parts that do not belong to pairs and parts that belong to matchedpairs. On the other hand, as shown at 1513, if it finds a member thatbelongs to the same class as the current target but is not the othermember of the pair, it puts the name of the other member of the pairinto the variable NewPict and exits loop 1509.

At loop 1515, the code deals with the situations where the part is not amember of a pair and where the part is a member of a matching pair. Inboth cases, the local variable NewPict is empty and loop 1515 randomlygenerates a number for a new face part. If the number is not that of thecurrent face part, loop 1515 exits, the name for the new face part ismade by adding the number to the current face part's name, and theresult is stored in NewPict (1516). In code portion 1517, finally, thegraphic that was thrown is replaced by the one specified in NewPict,with the replacement graphic being placed at the position on the displayindicated by OrigPosition.

Returning to FIG. 13, portion 1319 of on MoveMe 1215 implements resizingwhen the user touches the part twice. This portion of the code isexecuted after the user has ceased touching the part and has therebyterminated loop 1301 and after it has been determined that the user hasnot thrown the part. If the user touches the part again after a periodof 20 ticks has passed, the user-defined resizeMe event occurs. The codethat is executed on occurrence of the event is shown at 1401 in FIG. 14.

The code begins at 1404 by making a red outline around the part visible,indicating to the user that it has been selected for modification. Thebody of the code is a loop 1402 which is repeated while the usercontinues to touch the part. On each iteration of the loop, the currentposition being touched is saved in Position1, there is a pause of 8ticks, and the current position is saved in Position2. In the section ofthe code labeled 1405, the positions saved in the two variables arecompared to determine whether the motion of the touched point had avertical component. If the comparison indicates that the verticalcomponent was in the upward direction, the size of the part is increasedin the vertical direction (1407); if the vertical component of themotion was in the downward direction, the size of the part is decreasedin the vertical direction. Section 1411 works in the same fashion withany horizontal component of the motion. When the loop terminates, thedotted line showing that the part has been selected for modification isremoved.

Implementation of Other Interactions

It will be immediately apparent from the foregoing how the otherinteractions between the user and the child-friendly digital system areimplemented. In the case of the shape matching and puzzle assemblingactivities, the shapes are implemented as named foreground graphics andthe slots are implemented as named background graphics that have thesame name as the foreground graphics that corresponds to the slots. Theon MoveMe code that is executed when one of the named foregroundgraphics is touched includes dragging code similar to that justexplained, and while the dragging is going on, the code constantlychecks whether the present location of the foreground graphic is withina predetermined distance of the corresponding background graphic. If itis, the foreground graphic aligns itself with the background graphic.Moving a shape to its slot when the slot is touched is done with onMouseDown code. When the slot is touched, its name is used to constructthe name of the foreground graphic to be moved and the code then movesthe foreground graphic to the slot that has been touched.

In the display of pictures, the pictures are divided into categories,with the same number of pictures in each category. An array is createdfor the category names, and each category is thereby mapped to an index.Associated with each category name is an array for the pictures in thecategory, thereby mapping each picture to an index. A variable calledCurrCategory contains the index of the category of the picture currentlybeing displayed and another one called CurrPic contains the index of thepicture itself. A throw is detected from the motion of the touched pointin the display in the same manner as described above. When the throw isto the left, the current picture in the display is removed in thatdirection, the index in currPic is incremented by one, wrapping aroundif necessary, and the picture of the category that corresponds to thenew value of currPic appears. When the throw is to the right, the samething occurs, except that currPic is decremented.

When the throw is up, the current picture is removed, currCategory isincremented, wrapping around if necessary, and the picture correspondingto the value of currPic in the category indicated by the new value ofcurrCategory is displayed. When the throw is down, the same thingoccurs, except that currCategory is decremented.

CONCLUSION

In the foregoing Detailed Description, the inventor has described thebest mode presently known to him of adapting a digital system for use bychildren as young as two years in such fashion that those skilled in thearts to which the disclosure pertains may make and use such a digitalsystem. The child-friendly digital system disclosed herein has beenconstructed according to general principles including:

-   -   that the digital system be child-safe;    -   that a child can reach it easily;    -   that interactions with the digital system require neither the        use of keyboards or pointing devices such as a mouse;    -   that the system adapts itself to the child's skill level;    -   that interactions with the system do not require that the child        read;    -   that interactions do not require fine motor coordination; and    -   that interactions may be done directly on the objects displayed        on the screen.

One consequence of these principles is the physical form of thechild-friendly digital system: a system with no cords or otherappendages that sits on the floor and that uses a touch panel over anupward-facing display, a microphone, and position sensors for inputs andthe display and a loudspeaker for outputs. Another consequence is agraphical user interface which permits the child to manipulate objectson the screen by touching them and moving its finger on the screen. Theinterface is built from actions including touching an object to selectit for moving, moving the selected object by moving the finger acrossthe screen, moving the object at a speed faster than a thresholdvelocity to get rid of it, and modifying the object by touching it twiceand then moving the finger within the object to change its size. TheDetailed Description has further disclosed how these actions can be usedto create a number of activities, including shape sorting, assembling apuzzle, making a face out of component parts, and making what amounts toa digital picture book.

It will be immediately apparent to those skilled in the arts to whichthe disclosure pertains that there are many ways of realizing theprinciples demonstrated by the child-friendly digital system other thanthe one disclosed herein; it will also be apparent that many of theprinciples and techniques demonstrated in the child-friendly digitalsystem are useful in other situations where control of a computer systemby means of a keyboard and pointing device is difficult. For example,adults who have suffered a stroke or who are wearing heavy gloves havefine motor coordination problems comparable to those of small children.Moreover, interactions in the GUI such as throwing may be useful even insystems that employ standard pointing devices.

For all these reasons, the Detailed Description is to be regarded asbeing in all respects exemplary and not restrictive, and the breadth ofthe invention disclosed herein is to be determined not from the DetailedDescription, but rather from the claims as interpreted with the fullbreadth permitted by the patent laws.

What is claimed is:
 1. A system for manipulating images comprising ascreen upon which an image is displayed; and a computer coupled to thescreen, the computer causing the images to be manipulated in response tolocation inputs from a pointing device, the system being characterizedin that: when the image is being dragged in response to the locationinputs and the system detects that the velocity with which the image isbeing dragged exceeds a threshold velocity, the system responds byremoving the image from the display without leaving any representativethereof in the display.
 2. The system set forth in claim 1 wherein: theremoved image is automatically replaced by another image.
 3. The systemset forth in claim 2 wherein: there is a plurality images, each imagebelonging to a class of a plurality thereof according to the image'scontent; and when the image that is being removed is dragged in a firstdirection, the removed image is replaced with a different image of thesame class; and when the image that is being removed is dragged in asecond direction, the removed image is replaced with an image of adifferent class.
 4. The system set forth in claim 3 wherein: the classesbelong to a hierarchy; and the second direction includes a thirddirection specifying a class from a higher level in the hierarchy and afourth direction specifying a class from a lower level in the hierarchy.5. The system set forth in claim 1 wherein: the pointing device is atouch panel.
 6. The system set forth in claim 5 wherein: the touch panelis transparent and is mounted on the screen.
 7. A system formanipulating a movable image comprising: a touch-sensitive screen uponwhich an image is displayed; a computer coupled to the screen, thecomputer causing the image to be manipulated when the touch screen istouched, the system being characterized in that: touching the movableimage at a point within the image selects the image for moving and whenthe point being touched is being continually moved and the systemdetects that the velocity at which the point is moving exceeds apredetermined threshold velocity, the image being continually moved isremoved from the screen without leaving any representative thereof onthe screen.
 8. The system set forth in claim 7 wherein: the removedimage is automatically replaced by another image.
 9. The system setforth in claim 8 wherein: there is a plurality of images, each imagebelonging to a class of a plurality thereof according to the image'scontent; and when the image that has been removed was moving in a firstdirection, the removed image is replaced with a different image of thesame class; and when the image that has been removed was moving in asecond direction, the removed image is replaced with an image of adifferent class.
 10. The system set forth in claim 9 wherein: theclasses belong to a hierarchy; and the second direction includes a thirddirection specifying a class from a higher level in the hierarchy and afourth direction specifying a class from a lower level in the hierarchy.11. The system set forth in claim 9 wherein: touching the movable imageand touching the movable image again within a predetermined periodselects the movable image for modification.
 12. The system set forth inclaim 11 wherein: continually moving the point being touched after themovable image is again touched causes the image to be modified.
 13. Thesystem set forth in claim 12 wherein: continually moving the point beingtouched in a vertical direction changes the height of the image.
 14. Thesystem set forth in claim 12 wherein; continually moving the point beingtouched in a horizontal direction changes the width of the image. 15.Apparatus for displaying images, the apparatus comprising: a touchscreen for displaying the images and a computer coupled to the touchscreen, the computer responding to a continuing touch that moves theimage across the touch screen such that when the computer detects thatthe velocity of the touch exceeds a predetermined threshold, thecomputer responds by removing the image from the screen, and byreplacing the image with a replacement image that is not arepresentative of the removed image.
 16. The apparatus for displaying animage set forth in claim 15, wherein: there is a plurality of images,each image belonging to a class of a plurality thereof according to theimage's content; and when the continuing touch moves in a firstdirection, the replaced image is replaced with a replacement image ofthe same class; and when the continuing touch moves in a seconddirection, the replaced image is replaced with a replacement image of adifferent class.
 17. The apparatus set forth in claim 16 wherein: theclasses belong to a hierarchy; and the second direction includes a thirddirection specifying a class from a higher level in the hierarchy and afourth direction specifying a class from a lower level in the hierarchy.18. The system set forth in claim 2 wherein: the removed image and theother image belong to an ordered set thereof; if the removed image wasdragged in a first direction, the other image precedes the removed imagein the ordered set; and if the removed image was dragged in a seconddirection, the other image follows the removed image in the ordered set.19. The system set forth in claim 8 wherein: the removed image and theother image belong to an ordered set thereof; if the removed image wasmoved in a first direction, the other image precedes the removed imagein the ordered set; and if the removed image was moved in a seconddirection, the other image follows the removed image in the ordered set.20. The system set forth in claim 15 wherein: the removed image and thereplacement image belong to an ordered set thereof; if the removed imagewas moved in a first direction, the replacement image precedes theremoved image in the ordered set; and if the removed image was moved ina second direction, the replacement image follows the removed image inthe ordered set.